How to transform (d e normalize) predicted values when I normalize the data in the range between -1 to 1?
122848
Alright so i have found  really good answer on how to normalize my data.   I implemented @user25658 's code into my own project successfully, trained a linear model and used it to make a prediction.   Say y = 0.4513869 is the predicted normalized value which i would like to de-normalize.  Based on the normalization code below, how can i achieve this?  normalized = (x-min(x))/(max(x)-min(x))
74430
I'm currently trying to apply a linear model (family = gaussian) to an indicator of biodiversity that cannot take values lower than zero, is zero-inflated and is continuous. Values range from 0 to a little over 0.25. As a consequence, there is quite an obvious pattern in the residuals of the model that I haven't managed to get rid of:   Does anyone have any ideas on how to solve this?
42298
I have some data that is bounded between 0 and 1. I have used the betareg package in R to fit a regression model with the bounded data as the dependent variable. My question is: how do I interpret the coefficients from the regression?
99237
Here, have a look:  You can see exactly where the training data ends. Training data goes from $-1$ to $1$.  I used Keras and a 1-100-100-2 dense network with tanh activation. I calculate the result from two values, p and q as p / q. This way I can achive any size of number using only smaller than 1 values.  Please note I am still a beginner in this field, so go easy on me.
117954
I am having 'hour' field as my attribute, but it takes a cyclic values. How could I transform the feature to preserve the information like '23' and '0' hour are close not far.   One way I could think is to do transformation: min(h, 23-h)  Input: [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23]  Output: [0 1 2 3 4 5 6 7 8 9 10 11 11 10 9 8 7 6 5 4 3 2 1]   Is there any standard to handle such attributes?  Update: I will be using superviseed learning, to train random forest classifier!
58580
I am trying to produce a model for which I have a response variable which is a proportion between 0 and 1, this includes quite a few 0s and 1s but also many values in between. I am thinking about attempting a beta regression. The package I have found for R (betareg) only allows values  in between 0 and 1 but not including 0 or 1 them selves. I have read elsewhere that theoretically the beta distribution should be able to handle values of 0 or 1 but I do not know how to handle this in R.I have seen some people add 0.001 to the zeros and take 0.001 from the ones, but I am not sure this is a good idea?  Alternatively I could logit transform the response variable and use linear regression. In this case I have the same problem with the 0 and 1's which cannot be log transformed.
48392
I have a vector shapefile with a column with string values. How can I convert this to a column of integer values in another column but keep the categorization.  e.g. from:  col1,newcol2 A, A, B, A, C,    to:  col1,newcol2 A,1 A,1 B,2 A,1 C,3    I'm sure it is in the 'replace' or 'case when' syntax somewhere but I cannot find a clear method via Google etc.
27733
I have a data set in the form of (features, binary output 0 or 1), but 1 happens pretty rarely, so just by always predicting 0, I get accuracy between 70% and 90% (depending on the particular data I look at). The ML methods give me about the same accuracy, and I feel, there should be some standard methods to apply in this situation, that would improve the  accuracy over the obvious prediction rule.
55443
I've heard/seen in several places that you can transform the data set into something that is normal-distributed by taking the logarithm of each sample, calculate the confidence interval for the transformed data, and transform the confidence interval back using the inverse operation (e.g. raise 10 to the power of the lower and upper bounds, respectively, for $\log_{10}$).  However, I'm a bit suspicious of this method, simply because it doesn't work for the mean itself: $10^{\operatorname{mean}(\log_{10}(X))} \ne \operatorname{mean}(X)$  What is the correct way to do this? If it doesn't work for the mean itself, how can it possibly work for the confidence interval for the mean?
27828
Not sure if normalize is the correct word to use here, but I will try my best to illustrate what I am trying to ask. The estimator used here is least squares.  Suppose you have $y=\beta_0+\beta_1x_1$, you can center it around the mean by $y=\beta_0'+\beta_1x_1'$ where $\beta_0'=\beta_0+\beta_1\bar x_1$ and $x_1'=x-\bar x$, so that $\beta_0'$ no longer has any influence on estimating $\beta_1$.  By this I mean $\hat\beta_1$ in $y=\beta_1x_1'$ is equivalent to $\hat\beta_1$ in $y=\beta_0+\beta_1x_1$. We have reduced equation for easier least square calculation.  How do you apply this method in general? Now I have the model $y=\beta_1e^{x_1t}+\beta_2e^{x_2t}$, I am trying to reduce it to $y=\beta_1x'$.
46273
I am trying to work on the data of my master thesis. I have to create a model, but I don't know which analysis perform. My data look like this     number ID sp ab alt SWdiv     1 A1 4 6 630 0.25802965   2 A2 9 16 750 0.29235324   3 A3 8 14 850 0.411620203   4 A4 9 22 950 0.444079553   5 A5 2 12 1050 0.294855765   6 A6 13 40 1130 0.298451014   7 B1 2 2 750 0.275329152   8 B2 5 6 850 0.043352118   9 B3 14 16 950 0   10 B4 17 58 1050 0.230358688   11 B5 7 9 1150 0.298759725   12 B6 8 68 1250 0.012192474   13 C1 12 20 830 0.467542842   14 C2 13 19 930 0.454480722   15 C3 6 30 1030 0.464584038   16 C4 7 14 1130 0.448191833   17 C5 5 15 1250 0.052045275   18 C6 2 7 1315 0.051767368   19 D1 3 3 650 0.294855541   20 D2 11 17 750 0.291717599   21 D3 9 46 850 0.084284864   22 D4 8 57 950 0.088574421   23 D5 4 32 1050 0.160873283   24 D6 13 22 1150 0.297638962   25 D7 6 22 1250 0.290733158   26 E1 11 23 800 0.250878932   27 E2 8 8 900 0.418021272   28 E3 10 13 1000 0.132159303   29 E4 5 10 1100 0.058487554   30 E5 5 15 1200 0.250310822   31 E6 10 19 1350 0.01600807   32 F1 6 20 800 0.462497319   33 F2 15 54 900 0.376552904   34 F3 6 13 1000 0.298264791   35 F4 6 13 1100 0.394736812   36 G1 14 22 830 0.391455956   37 G2 11 16 930 0.282922551   38 G3 5 10 1030 0.282398234   39 G4 10 15 1130 0.153502726   40 G5 13 18 1230 0.051586706   41 H1 10 11 700 0.3009609   42 H2 7 14 800 0.579228102   43 H3 5 11 900 0.528108703   44 H4 20 48 1000 0.28689587   45 H5 13 18 1100 0.264542772   46 H6 10 20 1250 0.29928462   47 I1 12 13 535 0.35326876   48 I2 3 5 635 0.273947657   49 I3 18 30 720 0.467208221   50 I4 16 188 835 0.276669711   51 I5 7 69 935 0.000867067   52 L1 6 10 435 0.36910895   53 L2 7 20 606 0.236700524   54 L3 14 33 697 0.111132519   55 L4 17 42 797 0.296757045   56 L5 5 7 910 0.27539552   57 L6 10 15 1000 0.29748451   58 L7 6 11 1101 0.276755556     58 sample points each has two value: altitude and diversity index related to habitat heterogeneity. What I sample are different insects and I have two values: sp = species (total number of different species for each sample point) and ab = abundance (total number of insects for each sample point). My question are are the species richness or the abundance affected by the altitude or the habitat heterogeneity? which is the best analysis to perform? and why? thank you for your help.
92757
When performing the linear SVM classification, it is often helpful to  normalize the training data, for example by subtracting the mean and  dividing by the standard deviation, and afterwards scale the test data  with the mean and standard deviation of training data. Why this process  changes dramatically the classification performance?
100494
I would like to know how to transform negative values to Log(), since I have heteroskedastic data. I read that it works with the formula Log(x+1) but this doesn't work with my database and I continue getting NaNs as result. E.g. I get this Warning message (I didn't put my complete database because I think with one of my negative values is enough to show an example):   &gt; log(-1.27+1) [1] NaN Warning message: In log(-1.27 + 1) : NaNs produced &gt;    Thanks in advance  UPDATE:  Here is an histogram of my data. I'm working with palaeontological time series of chemical measurements, E.g the difference between variables like Ca and Zn is too big, then I need some type of data standardization, that is why I'm testing the log() function.   This is my
61628
I would like to convert several range of columns into rows, and keeping the association to the original col/row source  Sample source sheet  --------------------------------------------------- | Data 1 | 1/1/2018 | 1/2/2018 | 1/3/2018 | etc...365 days | --------------------------------------------------- | name 1  |    25    |   30     |    5     | etc...|  --------------------------------------------------- | name 2  |    2     |          |    4     | etc...| --------------------------------------------------- | etc ... 100 rows   Expected Results  --------------------------------------------------- | Data 1 |   Date      |  Value   |  --------------------------------------------------- | name 1  |  1/1/2018   |   25     |    --------------------------------------------------- | name 1  |  1/2/2018   |   30     |    --------------------------------------------------- | name 1  |  1/3/2018   |   5      |    --------------------------------------------------- | name 1  |  etc.. 365 days --------------------------------------------------- | name 2  |  1/1/2018   |   2      |    --------------------------------------------------- | name 2  |  1/2/2018   |          |    --------------------------------------------------- | name 2  |  1/3/2018   |   4      |    --------------------------------------------------- | name 2  |  etc.. 365 days --------------------------------------------------- | etc ... for each change of name pull all the col dates/values   Current results  --------------------------------------------------- | Data 1 |   Date      |  Value   |  --------------------------------------------------- | name 1  |  1/1/2018   |   25     |    --------------------------------------------------- |         |  1/2/2018   |   30     |    --------------------------------------------------- |         |  1/3/2018   |   5      |    --------------------------------------------------- |         |  etc.. 365 days                      I have to copy down to bring in "name 1" for each date  --------------------------------------------------- | name 2  |  1/1/2018   |   2      |             now update the formula to bring in the second item for the sequence of dates --------------------------------------------------- |         |  1/2/2018   |          |    --------------------------------------------------- |         |  1/3/2018   |   4      |    --------------------------------------------------- |         |  etc.. 365 days --------------------------------------------------- | etc ...     =Sheet1!A:A I can bring in the name from the source list, but I have to copy it down for the number of dates.  How can I have it match the number of dates?  and be dynamic to update as dates/values are added Edit: trying to use split and rept to count the number of times to repeat, then transpose.. =transpose(rept(split(arrayformula(concatenate(Sheet1!A:A&amp;"1")),"1",True, True),COUNTA(Sheet1!I1:ZZ1)))  =TRANSPOSE(Sheet1!$C$1:$Z$1) I can use transpose to get the dates/values, but it's fixed range.  How can I make it dynamic to import dates as they are added?
76844
Scientific talks have their own culture, reporting the results, typically for me, in a table of the following form  1.7333    1.6472    1.5418    1.4682    1.5740 1.1525    1.2878    1.2652    1.3511    1.5102 1.2181    1.1500    1.2220    1.2425    1.5699 0.9786    1.1051    1.1472    1.3096    1.6757 0.9569    0.9190    1.0433    1.2924    1.4867   There is clearly a(n almost perfectly) monotonic pattern for rows and columns. Now for the naked eye, this is not the most quick way of capturing informing.  I would like numbers that bigger in absolute value, to appear differently (either green vs red or light vs dark) than smaller numbers, such that the pattern becomes clear instantly.  Is there a way of programming this in LaTeX?
47919
For the LASSO (and other model selecting procedures) it is crucial to rescale the predictors.  The    is simply to use a 0 mean, 1 standard deviation normalization for continuous variables. But what is there to do with dummies?  E.g.  from the same (excellent) summer school I linked to rescales continuous variables to be between 0 and 1 (not great with outliers though), probably to be comparable to the dummies. But even that does not guarantee that the coefficients should be the same order of magnitude, and thus penalized similarly, the key reason for rescaling, no?
111848
I always read in books that when we do classification or machine learning tasks it's always better to normalize the features so to make them in one range like 0-1. Today I used weka to play with Iris dataset. First I just built a J48 classifier without normalizing the values, and the it made perfect performance. However when I normalized all the features to be in the range 0-1, the classifier made so much mistakes. Why is that? Shouldn't normalization be used always?